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ABSTRACT 

This report is an executive summary of a study 
conducted for the Minister's Advisory Committee on Student 
Achievement (KACOSA) of Alberta^ Canada. It was designed to 
investigate levels of student achievement in science in Alberta at 
grades three^ sixr nine and twelve and to provide a data base for 
t^iture assessment. Between 2,;000 and 3^000 students were tested at 
each grade level. The findings show that the student performances 
were generally satisfactory. Some areas of weakness were identified: 
(1) knowledge of scientific methods in grade three; (2) knowledge of 
physical science in grades three, six and twelve; and (3) earth-space 
science and general knowledge of science and scientists in grades 
nine anc twleve. A list cf recommendations is presented at the end of 
the report. (HM) 
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A STUDY CONDUCTED FOR 
THE MINISTER'S ADVISORY COMMITTEE ON 
STUDENT ACHIEVEMENT 



Executive Summary 



- by - 

Morris Treasure 
Planning and Research Branch 
Alberta Education 



Fall, 1978 



To speculate without facts is to attempt to 
enter a house of which one has not the key, by 
wandering aimlessly round and round, searching, 
the walls, now and then peeping through the 
windows. Facts are the key. 

- Julian Huxley 

Essays in Popular Science 



The Minister's Advisory Commi ttee on Student Achieve me nt 
(MACOSA) was established by ministerial order in October 1976 
in response to growing concerns expressed by the public-at- 
large, government, labor, business, students and educators 
regarding the quality and standards of basic education in 
Alberta. 

MACOSA commissioned a number of studies, primarily to pro- 
vide basic information for a summary of current levels of 
achievement in Alberta and to provide baseline data for future 
assessment. These studies fell into three categories: (1) 
preliminary studies^ (2) achievement studies, and (3) other 
s tudles • 

This achievement study. Alberta Science Achievement 
S t udy , was designed to provide information about current 
levels of achievement in science among students in Alberta 
schools and to provide a data base for future assessments. 

This report, which represents the findings and conclusions 
of the researcher, was presented to MACOSA as information. 
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Science Achievement Study 



Purposes 

The purposes of this study, entitled the Alberta Science 
Achievement Study , were: 

!• To Investigate current levels of student achievement In 
science In Alberta at the grades three, six, nine and 
twelve levels . 

2. To provide a data base for future assessments. 



Procedures 

For the grade three test the researcher chose 114 Items 
from a variety of sources, and a randomly selected group of 20 
primary teachers validated the Items. For the grade six test 
the researcher chose 144 Items from similar sources, and a 
panel of 18 upper elementary teachers validated them. A 
determined effort was made to select Items which would reflect 
the objectives of the elementary science program. There are 
six content areas In this program--two In physical science, 
three In biological science and one In earth science. Inter- 
woven with these content areas are objectives related to the 
methods of science, and attitudes toward and knowledgji of sci** 
ence and scientists. 

The tests for grades nine and twelve were made up of two 
published standardized tests: the Sequential Tests of Educa- 
tional Progress (STEP), Series II, and Test of Understanding 
Science (TOUS), Form Jw. The STEP II measures student 
achievement levels and the TOUS measures students' knowledge 
of and opinions about scientists and science as a field of 
study. For grade nine the researcher chose the STEP II, Form 
3A (with the addition of 15 supplementary Items) because It 
best matched the Alberta curriculum. In grade twelve, where 
the emphasis was to be on more general knc^wledge of science, 
both in terms of the content of the high school program and 
the general alms and purposes of science, the researcher chose 
the STEP II, Form 2A, because of its relative quality and 
suitability for gathering the required information. The TOUS, 
Form Jw was the only appropriate attitude test among the few 
such tests which are available. 

The Student Evaluation and Data Processing Branch of 
Alberta Education selected a stratified random sample of 
schools consisting of 101 schools offering grade three; 96 
schools offering grade six; 40 schools offering grade nine; 
and 24 schools offering grade twelve. There were 3073 grade 
three students 2935 grade six students, 2426 grade nine stu- 
dents, and 2125 grade twelve students tested. These represent 
about 8 percent of the total number of students enrolled in 
thesegrades. 
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All tests were administered on May 17, 1978, 
Findings 

Table 1 shows the levels of performance for each grade 
level and general content area, reported as percentage of 
correct responses, and the number of items used to test each 
area. 

TABLE 1 



Summary of Science Achievement Levels: 
Percentage of Correct Responses 



Content Area 






Grade 








6 


9 


1 2 


Physical Science 


55.2 


59.6 


62.6 


55.5 


(21)* 


(38) 


(17) 


(32) 


Biological and 


63.7 


61.8 


69.3 


72.7 


Life Science 


(26) 


(45) 


(12) 


(16) 


Earth Science 


77.2 


70.8 


56.7 


47.9 




(6) 


(11) 


(17) 


(3) 


Methods of Science 


57.4 


64.9 


6 2.^* 


64.8 




(41) 


(24) 


(19) 


(36) 


Average Student 










Achievement in 










Content Areas 


60.3 


62.5 


62.2 


62.2 


Science as a Human 


60.6 


71.1 


49.2 


57.3 


Endeavor 


(14) 


(14) 


(45) 


(45) 


Interest in Science 


66.1 


61.2 








(6) 


(12) 







Number of test items is shown in parentheses. 



In grade three the performance levels were highest on 
earth science, even though the curriculum does not stress 
earth science; and lowest on* physical science, which includes 
a number of difficult concepts such as electricity, molecules 
and basic energy conservation. Grade three students regis- 
tered a relatively low level of achievement in methods of sci 
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ence investigation despite the fact that science programs have 
emphasized this area over the past few years. These students 
demonstrated a reasonable amount of knowledge in the area of 
science as a human endeavor, a section of the test designed to 
check students' perceptions of science and scientists. 

The grade six curriculum has recently emphasized physical 
science but, even so, grade six scores in physical science 
were not high. Scores were high on biological science and 
life science items and as in grade three, students at the 
grade six level had high scores on the earth science items. 
Reli?itively high scores on science methods items probably 
reflect an increased curricular emphasis on this topic. In 
grade six scores on perceptions of science and scientists we re 
also relatively high. 

Grade nine students achieved the highest average scores on 
items dealing with biological and life science, and the next 
highest average score on physical science. The achievement 
level in earth science was quite low. Scores were high on 
methods of science while "science as a human endeavor", which 
included such topics as aims of science and the skills and 
aptitudes of scientists, had a low response rate. 

Grade twelve students achieved at the highest level on 
biological and life science items and lowest on earth science 
items. Performances were relatively high on methods of sci- 
ence and relatively low on science as a field of human 
endeavor • 

The tests for grades three and six contained 47 common 
items, including 18 items drawn from the content areas for 
grade three, 18 items drawn from grade six content areas and a 
further 11 items measuring interest in science and opinions 
and beliefs about scientists. Table 2 indicates relative stu- 
dent performance on common items. 





TABLE 2 








Performance on Common Items 
Percentage of Correct 


on Science 
Responses 


Tests : 






Student 
Grade 3 


Performance (%) 
Grade 6 


Grade 3 


Target Items (18) 


63.0 


77.0 


Grade 6 


Target Items (18) 


49.4 


59. 1 


Interest 


and Opinion Items (11) 


61.4 


74.0 


TOTAL 




57.4 


69.4 



,9 
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As expected, grade six student performance levels on the 
common Itemswere substantially higher than grade three per- 
formance levels. However, the grade six performance on grade 
six Items was somewhat low. 

Relative performance on common Items by content area, as 
shown In Table 3, Is not as consistent. 



TABLE 3 



Performance on Common Items 
on Science Tests by Content Area: 
Percentage of Correct Responses 



Content 


Student 
Grade 3 


Performance (% ) 
Grade 6 


Physical Science (5)* 


62.9 


64.3 


Biological and Life 
Science (10) 


58. 2 


74.2 


Earth Science (4) 


73. 1 


86.9 


Methods of Science (13) 


48.4 


58. 1 


Science as a Human 
Endeavor (10) 


57.5 


74.9 


Interest in Science (5) 


61. 1 


70.0 


* Number of test items is 


shown in parentheses 





The most substantial differences between performances by 
grade thr\e and grade six on common items occurred on item 
clusters dealing with earth science, biological and life sci- 
ence, and science as a human endeavor (opinions and beliefs 
about scientlRts), while smaller differences occurred on item 
clusters dealing with physi cal science, methods of science and 
interest in science. 

Because both grades nine and twelve students responded to 
the TOUS, Form Jw, and to different forms of the STEP II, the 
researcher was able to make similar comparisons between these 
grades. Table 4 compares student performances in grade nine 
and twelve on the STEP II, for Alberta and the United States. 
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TABLE 4 



STEP 11 Test Results: 
Average Raw Scores 





Form 3 A 


(Grade 9) 


Form 2A 


(Grade 12) 




Albert'.a 


U.S.A. 


Albert a 


U. S. A 


(Spring '78) 


(Spring '70) 


(Spring '78) 


(Spring'70) 


N 


2426 


2637 


2125 


2285 


Average 


34 


32 


4 7 . 


42 


Score 


(50) 


(50) 


(75) 


(75) 


Standard 










Devlat Ion 


7 


12 


12 


13 



Total number of test items is shown in parentheses. 



The average response of grade nine students on the STEP II 
was 34 out of 50 items, as compared with 32 for their American 
counterparts. The average score for the grade twelve students 
was 47 out of 75 items, as compared with 42 for the American 
norming sample. At both grade levels the high Alberta per- 
formances are statistically significant (p < 0.05). 

Table 5 compares performances by students in grades nine 
and twelve with the results further broken down to show per- 
formance levels for girls and boys. 

TABLE 5 



TOUS, Form Jw , Test Results: 
Average Raw Scores 





Grade 9 


Boys 


Girls 


Grade 12 


Boys 


Girls 


N 


2426 


1234 


1 144 


2125 


1059 


1034 


A v e r a g e 
Score 


21.8 

(45)* 


21.1 


22.6 


25.3 
(45)* 


24.6 


25.8 


Standard 
Deviat ion 


6.0 


6.1 


5.8 


6.5 


6.7 


6.2 


* Total number 


of test 


items 


is shown 


in parentheses. 
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The TOUS has not been used in Alberta long enough to make 
comparisons other than those shown In Table 5. As one might 
expect^ grade twelve students scored higher than grade nine 
students. In both grades nine and twelve, the girls outper- 
formed the boys. This result suggests that girls understand 
Che nature of science better than do boys. 

An examination of grades nine and twelve results by 
content area (not shown In table form) Indicated that both 
grou::s performed at a high level on the biological and life 
science Items and at a fairly low level on Items asking about 
the nature of science (Interest and opinion Items). Both 
grades performed above the test average on Items related to 
scientific methods, and average performance on physical sci- 
ence Items was above the median for grade nine but below the 
median for grade twelve. 

The researcher also categorized the test Items by the 
thought level required for giving the correct response. Table 
6 dhows the average performance for three thought levels-- 
knowledge, comprehension and application. 



TABLE 6 



Average Performance by Thought Level of Items: 
Percentage of Correct Responses 



Grade 


Knowledge 


Thought Level 
Comprehension 


Application 


3 


68.3 (16)* 


57.8 (24) 


67.0 (15) 


6 


67.4 (38) 


57.7 (32) 


59.3 (26) 


9 


64.7 (13) 


69.6 (18) 


58.6 (25) 


1 2 


61.5 (15) 


63.1 (15) 


58.6 (20) 


* Number of 


Items Is shown 


In parentheses. 





Elementary students had higher scores than secondary stu- 
dents at the knowledge level, but lower scores than secondary 
students on comprehension Items. Scores on Items at the 
application level were substantially lower than scores at the 
knowl edge level for all grades except grade three. 



Conclusions 

These data show the present level of student achievement 
in Alberta science programs on a number of dimensions. 




An exammacion or graaes nine ana cwexve results Dy 
content area (not shown In table form) Indicated that both 
grou:^s performed at a high level on the biological and life 
science Items and at a fairly low level on Items asking about 
the nature of science (Interest and opinion Items). Both 
grades performed above the test average on Items related to 
scientific methods, and average performance on physical sci- 
ence Items was above the median for grade nine but below the 
median for grade twelve. 

The researcher also categorized the test Items by the 
thought level required for giving the correct response. Table 
6 dhows the average performance for three thought levels-- 
knowledge, comprehension and application. 

TABLE 6 



Average Performance by Thought Level of Items: 
Percentage of Correct Responses 



Grade 


Knowledge 


Thought Level 
Comprehension 


Application 


3 


68.3 (16)* 


57.8 (24) 


67.0 (15) 


6 


67.4 (38) 


57.7 (32) 


59.3 (26) 


9 


64.7 (13) 


69.6 (18) 


58.6 (25) 


1 2 


61.5 (15) 


63.1 (15) 


58.6 (20) 


* Number of 


Items Is shown 


In parentheses. 





Elementary students had higher scores than secondary stu- 
dents at the knowledge level, but lower scores than secondary 
students on comprehension Items. Scores on Items at the 
application level were substantially lower than scores at the 
knowledge level for all grades except grade three. 



Conclusions 

These data show the present level of student achievement 
In Alberta science programs on a number of dimensions. 
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Whether this level is good, bad ^^tt^xag^ J^^y difficult 
to judge. The descriptive ^nfor^^Xi^^K is by this 

study to provide a basis for c^^^J^^a^ > ^^hi data points 

for some future assessment. Th® of ^ ^ ^tudy be 

determined at that future date* 

The lower performance of the ^ thi^^^ ^^^^^ent^ on the 

physical science items was thot^S^^h^^ be ^.'^^^ble t^^caus© it 
has been recognized that both *^®^^hig^ and ^f^^^ts h^^e 
experienced some difficulty with /f^^ ar^^ ^ cuf^^culum. 
The strong performances in both ^^isf^eieti^^^^^^^ earth* space 
science were judged to be very ^^^rj ^ing' ^^uld ^® 

hypothesized that these student f ^a^i^^^anc^^^^^ft influenced 
by the relative emphasis on the ^^s ^ theto^ ^Hildf^n's 
television programs and the effojT p^°? qwc^ ^^^^tXizat ions 
National Geographic in producing ^^als life ^^^^ence 

theme . 

The only area in elementary ^^t^^^e th^^^^^ to be some- 
what unsatisfactory was the st ude^p^^^^^f oi^^^£^^ On items 
related to science methods. Thi^^f ^^er o^^^^^^ level is 

probably due to a different view t rol^ Student activ- 

ity in science* Otherwise, stude^i^P^^for*^f|'^^^ Was g^^^^aHy 
satisfactory in the content ^i-^^^^f^^^ of ^^Ogram- 
However, the secondary student $ ^a^nc^ ^ h it^^^ 
related to the earth-space sci^nc^o^^^^ ^^^ela somewhat 
inadequate. One possible reason j ^hls ^ "^^ely Poor stu- 
dent performance could be the jeV^ ab^t^^^^^^On of ^^e con- 
cepts. 

There is some concern about t^^^f^^qixa^^ ^tud^nt per- 
formance on items related to th® ^^4e^^^ a j^^tists- Stu- 
dent performance levels at both ^^e^ ^ ^^^^ a " ^wel^^^ were 
above the test average, so the^e VieW^^ ^ ^ati^^^^^Ofy . 

The program objectives related ^o.^nj^^^ af^^ Qfteti neither 

accepted nor understood as beltig d^^^^ati^ ' "^^her ^^ea of 
weak student performance was tti ^^Utx^j^ t and i^^^ nature of 
science, as tested by the Test of of*.^^^*'^^ Science. This 

area is difficult to teach and is vl^^^ bei^S peri- 

pheral to the main intent of sde^ ^bog*^^^^* 

In general terms, the stud^nt^^^^^ rma^^^^ ^ix grades three 
and six were satisfactory. Aj^o jeX^^^e^^^ ^^^^^ th^ student 
performances at grades nine and ^ ^fti we^^ ^^^^^facto^y with 
the exception of the junior high ^ gci^^^^ ^^ea. 

A further measure of adequacy ^j/ai ^^V-^ the second- 

ary level because of the use of 3 

Z^^tAi^...^^^t. with U.S. 
norms. The average student Peffo^the^^ T S grades 

nine and twelve was above that iexA I 970 * * ^Xorms • Since 
1970 there has been a documented i^f ^ ^^^ardi^^^i test 

scores across the U.S., so thj^ ^^o^^^nce ^^^^^^ts a niuch 
greater difference in performative ^ ^'^^^^ mo^e 

recent norms were available. i/si^ ^^^g ^ ^^^ndar^t stu- 

is 



dent performance in the field of secondary science in Alberta 
is satisfactory. 

The investigation therefore concluded that, on balance, 
student achievement is satisfactory. A number of weak areas 
have been identified, and there are a number of areas in which 
student performance is quite strong. 

The areas of some weakness include physical science in 
grades three and six, earth science in grade nine, and the 
grades nine the twelve responses to items asking about the 
purposes and aims of science as a field of human activity. 

Areas in which there is some degree of satisfaction are 
those dealing with the methods and strategies of science and 
the life sciences. 



1 . 1 0 Recommendat ion 

The invest igator recommends: 



l.lO.l that assessment of the science program in Alberta be 
a continuing process using the appropriate data from 
the present study as a base-line against which future 
achievement levels can be evaluated. 



The main value of undertaking an assessment of student 
achievement at this time lies in the use of the data as a 
point of comparison for future assessments. The present 
assessment efforts should represent a beginning point in the 
on-going evaluation of the education program in Alberta 
schools . 

1.10.2 that computerized item banking be developed and main- 
tained. 



It is recognized that the instrumentation developed and/or 
purchased for this assessment was not wholly satisfactory and 
that further work on many of the items needed to make them 
fully acceptable measures of achievement. The establishment 
of a computerized item bank would facilitate this process by 
simplifying access to the items and the accumulation of data 
about the items . 

When an acceptable bank of items categorized according to 
program objectives is available, tests may easily be con- 
structed to measure student achievement along a number of pro- 
gram dimensions. This data can then be accumulated to compile 
annual or biennial reports. By making the bank available to 
schools and teachers, the qu'ality of testing in the province 
can be improved. The teacher gains access to a bank of proven 
items, and the banked items can be improved and extended by 
using input from teachers to develop and improve them. 
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Information about student performance on specific curric- 
ular objectives could be made available to curriculum devel- 
opers on short notice. There could be the capability to res- 
pond very quickly to requests for information from those with 
a legitimate need. 

1.10.3 that a computerized bank of items used for the grades 
three and six testing be extended and improved. 

The items used at the elementary level were proven items 
from other assessment programs which measured achievement of a 
few selected objectives. Coverage of the program objectives 
should be broadened by the inclusion of more items. In gen- 
eral, the items were technically adequate but revision of some 
items could make them more applicable to the Alberta program. 

1.10.4 that computerized bank of items be developed for 
grades nine and twelve. 

To expedite the administration of the testing program, a 
commercially developed test was used to collect the base-line 
data. This provided some valuable insights into levels of 
student achievment in Alberta. But as with any instrument 
developed to sample the broad domain of program objectives, 
there are a few problems with making inferences about the 
quality of student achievement in Alberta. If an item bank 
were developed for Alberta, a better match between the test 
and the curriculum could be achieved. 

A start on a bank of Alber ta- valid items could be madfi by 
including items from former grade nine depar tmentals , other 
assessment programs, and locally-developed tests. 

The grade twelve item bank should focus less on the con- 
tent dimension and more on the practical level of scientific 
and technical knowledge and applications that should be 
expected of all graduates of the Alberta education system. 

1.10.5 that the scope of the items in an item bank should 
continue to include the full spectrum of program 
objectives, including those in the effective domain. 

The exclusion of any particular program objectives may 
well infer that a lesser importance is placed on such objec- 
tives. It would be unfortunate if the weighting of curricular 
objectives came as a result of an unplanned and unconscious 
distribution of items. Student attitude towards science is 
another objective that should not be neglected. 

1.10.6 that standard4^ed tests of educational progress 
should continue to be administered in order to make 
comparisons beyond provincial boundaries. 

It is too easy to become parochial and narrowly focussed 
in our view of program. To counter this tendency, a regular 

io 
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sampling of student achievement should be instituted as part 
of the continuing process of program evaluation. 

1.10.7 that the provincial testing program be conducted at a 
time of year which is, most convenient. 

Separation of the provincial assessment program from 
school-based student evaluation procedures v^ould serve to 
lessenthe impact of over-testing in the year-end period. In 
situations where it is reasonable to return test results to 
the teacher, it would be at a time when the teacher can use 
such results to modify instruction. 

1.10.8 that an item validation procedure be instituted to 
capitalize on the lessons learned from the Re suits 
Interpretation Panels. 

The use of a broadly based reaction panel for the purpose 
of validating the content of the test items for use in either 
a bank or a provincially administered test is recommended. 
Further involvement of a commupi ty-based group of individuals 
would serve to widen the scof<' ' : provincial evaluations 
beyond the rather narrow view /eld by some curriculum special- 
ists. If the aim of schooling is to produce an "educated" 
student, in the broader sense of community expectation, then 
representatives of that community should be involved in a 
reaction role at the initial steps of the evaluation and not 
just at the final stage. 

1.10.9 that a longitudinal study of a particular group of 
students (cohort) along a few broad curricular goals 
such as the scientific process be undertaken to 
investigate the grade placement of the spec'ific 
objectives and their match with student compe- 
tencies. 



Much of present science program has evolved as a result of 
experiences elsewhere and the availability of certain pub- 
lishers' materials. It would be both logical and beneficial 
to gather information about the present nature of our science 
program before we expend resources on developing program 
materials or strategies to change the science program. The 
information gleaned from such a study would be of great bene- 
fit to the re vision committees. 



1.10.10 that a further analyses of the data collected in this 
assessment be undertaken by qualified researchers 
either within Alberta Education or in the educational 
communi t y . 

The anonymity of students and schools in the study would 
be fully protected, but there are many cross-correlations and 
factor analyses which were not made and which could in fact 
lead to an improved assessment. 

^6" 
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1.10.11 that a study of the effect of variables such as size- 
of school, presence (or absence) of laboratories, and 
amount of time spent on science be commissioned by 
Alberta Education. 

The present study was limited in its scope by the very 
nature of the questions to be answered. However^^ this kind of 
process-product information is of value in examining some of 
the reasons for differences in student performance. 

1.10.12 that the testing procedures using matrix-sampling be 
continued in future province-wide assessments of stu- 
dent performance. 

The matrix-sampling procedure proved to be a very effi- 
cient, economical way of gathering valid data from across the 
province on a broad spectrum of objectives. One has to recog- 
nize that although a provitvcial testing program has little 
attraction for either teachers or students, the cooperation of 
these two groups is vital to its success. Therefore, limiting 
student time and minimizing the effort required of the teacher 
pays off in a greater degree of cooperation. From a provin- 
cial perspective, matrix sampling was facilitative. However, 
from a local perspective (if feedback is to be provided 
locally about individual pupils) it might not be applied. The 
purpose for the testing will dictate the appropriate sampling 
procedures • 
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